Apparatus and method for reducing image blur in a digital camera

ABSTRACT

A digital camera delays the capture of a digital image after image capture has been requested until the motion of the digital camera satisfies a motion criterion. The digital camera thereby reduces image blur that would otherwise occur due to camera motion.

FIELD OF THE INVENTION

The present invention relates generally to digital photography and morespecifically to apparatuses and techniques for reducing image blur in adigital camera.

BACKGROUND OF THE INVENTION

A pervasive problem in photography is blur due to camera motion. Somefilm cameras and other optical devices such as binoculars include highlysophisticated active image stabilization systems that deflect the imagepath slightly in a direction opposite of the camera motion. Such activestabilization systems are, however, both complex and expensive.

One alternative is to use a faster lens. Digital cameras already use thefastest lens practical in terms of cost, size, and desired imagequality. Lenses with maximum apertures of f/2 to f/2.8 are typical.Still faster lenses are much more expensive and bulky.

It is thus apparent that there is a need in the art for a digital camerathat reduces image blur without recourse to expensive or otherwiseimpractical solutions.

SUMMARY OF THE INVENTION

A method for reducing image blur in a digital camera by the tracking ofcamera motion is provided. An apparatus for carrying out the method isalso provided.

Other aspects and advantages of the present invention will becomeapparent from the following detailed description, taken in conjunctionwith the accompanying drawings, illustrating by way of example theprinciples of the invention.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A is a functional block diagram of a digital camera in accordancewith an illustrative embodiment of the invention.

FIG. 1B is a conceptual diagram of the motion management logic shown in

FIG. 1A in accordance with an illustrative embodiment of the invention.

FIG. 1C is a circuit diagram of an input control in accordance with anillustrative embodiment of the invention.

FIG. 2A is an illustration of a digital preview frame, of which acentral portion is a motion measurement region, in accordance with anillustrative embodiment of the invention.

FIG. 2B is an illustration of a digital preview frame, of which aperipheral portion is a motion measurement region, in accordance with anillustrative embodiment of the invention

FIG. 2C is an illustration of separate horizontal and vertical sets ofpicture elements that may be used in measuring the motion of a digitalcamera in accordance with an illustrative embodiment of the invention.

FIG. 3A is an illustration of a movement trajectory of a digital camerain accordance with an illustrative embodiment of the invention.

FIG. 3B is an illustrative plot of the magnitude of composite cameramotion annotated with time and threshold parameters in accordance withan illustrative embodiment of the invention.

FIG. 4 is a flowchart of the operation of the digital camera shown inFIG. 1A in accordance with an illustrative embodiment of the invention.

FIG. 5 is a flowchart of the operation of the digital camera shown inFIG. 1A in accordance with another illustrative embodiment of theinvention.

DETAILED DESCRIPTION OF THE INVENTION

Camera motion or “shake” tends to be somewhat periodic in both thehorizontal and vertical directions. At instants of direction reversal,the camera is nearly stationary, just as a child's swing is momentarilystationary at the instant it reaches the full extent of its travel ineither direction. Image blur in a digital camera may therefore bereduced by delaying capture of a digital image, after actuation of theshutter release button, until the motion of the digital camera satisfiesa motion criterion. For example, image capture may be delayed untilcamera motion reaches one of those instants of minimal motion (a localminimum). To avoid an unacceptable lag following actuation of theshutter release button, the delay may be constrained not to exceed apredetermined timeout period, or other criteria may be used to capture adigital image despite the motion criterion not being satisfied.Implementation of this technique requires some method for measuring themotion of the digital camera in approximately real time. Motionestimation algorithms may be relatively simple or quite complex. Oneexample of sophisticated motion estimation well known in the videoencoding art is that implemented in connection with the Moving PicturesExpert Group (MPEG) video compression standards.

FIG. 1A is a block diagram of a digital camera 100 in accordance with anillustrative embodiment of the invention. In FIG. 1A, controller 105communicates over data bus 110 with imaging module 115, input control120, display 125, motion management logic 130, timer 135, and memory140. Memory 140 further comprises random-access memory (RAM) 145 andnon-volatile memory 150. Optical system 155 produces optical images thatare converted to digital images by imaging module 115. Optical system155 may comprise, for example, a zoom lens. Imaging module 115 maycomprise an array of photosensors based on charge-coupled-device (CCD)or CMOS technology, an analog-to-digital converter (A/D), a gaincontrol, and a digital signal processor (DSP) (not shown in FIG. 1A).Imaging module 115 may be operated in a video preview mode in whichdigital preview frames are acquired at a rate of, for example, 30 framesper second and shown on display 125. Digital camera 100 may operate inthis video preview mode while operations such as autofocus,autoexposure, and motion tracking are performed. A CMOS photosensorarray has the advantage that pixels can be addressed directly like RAM,which simplifies and speeds the readout of the image data for theseoperations.

Motion management logic 130 may comprise hardware, firmware, software,or a combination thereof. Motion management logic 130 may beconceptualized as having two aspects: motion measurement logic 132 andcontrol logic 134, as shown in FIG. 1B. Motion measurement logic 132performs motion estimation on digital preview frames obtained fromimaging module 115 during video preview mode. These digital previewframes may be of a lower resolution than a final digital image tofacilitate video preview mode. Control logic 134 analyzes motionestimation information obtained from motion measurement logic 132 todetermine when a digital image should be captured after capture of animage has been requested. In one illustrative embodiment, controller 105comprises a microprocessor, and motion management logic 130 comprisesstored program instructions in software or firmware or a combinationthereof that may be executed by controller 105. In such an illustrativeembodiment, the combination of controller 105, imaging module 115, andmotion measurement logic 132 may be termed, functionally, a motiontracking subsystem that outputs an indication of the motion of digitalcamera 100 as a function of time. Controller 105, in accordance withcontrol logic 134, selects the instant of image capture based on theoutput of the motion tracking subsystem.

FIG. 1C is a circuit diagram of input control 120 in accordance with anillustrative embodiment of the invention. In FIG. 1C, shutter releasebutton 160 is capable of actuating, sequentially, switches S1 165 and S2170. When shutter release button 160 is partially depressed, switch S1165 is closed. When shutter release button 160 is further depressed,switch S2 170 is also closed. Prior to the closing of switches S1 165and S2 170, signals 175 and 180, respectively, are both in a logic“high” state. Signals 175 and 180 are connected with data bus 110. Thelogic “high” state is provided by connection of the switches between acommon ground and a positive voltage +V across pull-up resistors 185 and190. When switch S1 165 is closed, the corresponding signal 175 ispulled down to ground potential, generating a logic “low.” Likewise,when switch S2 170 is closed, the corresponding signal 180 is pulleddown to ground potential, generating a logic “low.”

Input control 120 may be used to trigger multiple operations in digitalcamera 100. For example, actuation of switch S1 165 may activateautofocus and autoexposure. Once autofocus and autoexposure adjustmentsare complete, a motion-tracking mode may be activated in which themotion of digital camera 100 is tracked. Actuation of S2 170 may signala request that a digital image be captured and stored. In a prior-artdigital camera, such capture would be immediate (without intentionaldelay). To minimize image blur caused by camera motion, however, it isadvantageous to delay capture of the digital image until a moment whenthe motion of digital camera 100 is at an approximate local minimum. Inother embodiments, input control 120 may include only one switch insteadof two. In those embodiments, a single signal from input control 120 mayrequest the capture of a digital image, and motion tracking may beactivated by a separate input signal (e.g., the power of digital camera100 being turned on) or by the same single signal from input control120.

Measuring the motion of digital camera 100 may be implemented in avariety of ways. The most obvious is through the use of motion sensors,such as accelerometers or gyroscopes. In better keeping with the lowcost and complexity objectives of the invention, motion can be measuredusing the image sensor itself. Motion estimation algorithms are wellknown in the video encoding art. Motion estimation generally involvescomparing at least one picture element (pixel) in a first frame with atleast one pixel in a second frame to discern a change in the sceneduring the interval between the two frames. This process may be repeatedfor successive pairs of frames to track camera motion relative to thebackground of the scene in approximately real time. In the context ofthe instant invention, motion estimation may be performed on digitalpreview frames obtained in the video preview mode of digital camera 100.

The comparison of pixels may also be implemented in a variety of ways.For example, the magnitude of the pixel-by-pixel difference inbrightness (luminance) may be computed. Alternatively, a pixel-by-pixelcorrelation (multiplication) may be performed. If the pixels comparedare in corresponding locations in the two digital preview frames, anindication may be obtained that motion of some sort between the framesoccurred but not how much or in what direction. For this reason, motionestimation techniques typically also include a search algorithm in whichone or more groups of pixels in a first digital preview frame arecompared with groups of pixels within a predetermined search regionsurrounding each corresponding location in a second digital previewframe. A motion estimation algorithm typically computes a motion vectorindicating the magnitude and direction of motion during a particularinterval. This motion vector may be expressed conveniently as horizontaland vertical motion components.

Sophisticated motion estimation techniques used in connection with MPEGcompression may improve the performance of motion estimation. Suchimprovements may include, for example, a fast search algorithm or anefficient computational scheme in addition to the general methoddescribed above. Such methods are well known in the video encoding art.One example of sophisticated MPEG motion estimation may be found in U.S.Pat. No. 6,480,629, the disclosure of which is incorporated herein byreference.

FIG. 2A depicts a digital preview frame 205 in accordance with anillustrative embodiment of the invention. Motion estimation may beperformed using one or more pixels within motion measurement region 210(cross-hatched in FIG. 2A). In FIG. 2A, motion measurement region 210comprises a central portion of digital preview frame 205. Such a regionmay coincide with the region used in performing autofocus orautoexposure. In such an embodiment, motion estimation may sharevirtually the same video preview mode of digital camera 100 withautofocus and autoexposure. One disadvantage of this approach, however,is that a moving subject within the central portion of digital previewframe 205 may be detected instead of the motion of digital camera 100relative to the background.

FIG. 2B shows one method for overcoming the problem of subject motion inaccordance with an illustrative embodiment of the invention. In FIG. 2B,motion measurement region 210 comprises a peripheral portion of digitalpreview frame 205 where an important subject is less likely to be found.By confining motion estimation to the periphery, subject motion may beexcluded, allowing the motion of digital camera 100 relative to thebackground of the scene to be measured.

In performing autofocus, digital cameras often apply a window functionat the boundary of the autofocus region to minimize edge effects causedby contrasty image data at the boundary. The window function attenuatesthe edges of the autofocus region in a tapered fashion, resulting in a“soft” boundary. Such window functions are well known in the digitalcamera art. A window function applied at the boundary 215 delineatingmotion measurement region 210 may be advantageous for the same reason.

FIG. 2C is an illustration of separate horizontal and vertical sets ofpixels that may be used in performing motion estimation in accordancewith an illustrative embodiment of the invention. In FIG. 2C, horizontalsets of pixels 220 and vertical sets of pixels 225, both lying withinperipheral motion measurement region 210, may be used in performingmotion estimation, as described above. Horizontal and vertical sets ofpixels 220 and 225, respectively, may be single rows or columns ofpixels or “strips” of pixels comprising multiple rows or columns. Oneadvantage of this approach is that measurement of horizontal andvertical motion may be separated into two sets of computations (e.g.,difference or correlation), each set of computations producing anestimated motion component. The limited number of pixels involved mayalso simplify the search algorithm. The choice of pixel sets shown inFIG. 2C is only one possibility of many. Fewer or more than the foursets of horizontal and vertical sets of pixels shown in FIG. 2C may beincluded in motion estimation.

FIG. 3A is an illustration of a movement trajectory of a digital camerain accordance with an illustrative embodiment of the invention.Trajectory 305 in FIG. 3A depicts the path of movement made by digitalcamera 100 during an arbitrary period prior to an image being captured.Local minima 310, where the motion of digital camera 100 changesdirection, are circled. Capturing a digital image at one of these localminima 310 may reduce image blur. Since the motion of digital camera 100may not reach a local minimum in the horizontal direction at the sameinstance it reaches a local minimum in the vertical direction, this mustbe taken into account in designing criteria for image capture.

Many possible quantities may be chosen as the output of the motiontracking subsystem. If motion measurement logic 132 measures horizontaland vertical motion components (e.g., velocities), one possible choiceis the square root of the sum of the horizontal motion component squaredand the vertical motion component squared (magnitude of the motionvector). Another possible choice is the sum of the absolute value of thehorizontal motion component and the absolute value of the verticalmotion component.

FIG. 3B is an illustrative plot 315 of the magnitude of composite motionof digital camera 100 annotated with time and threshold parameters inaccordance with an illustrative embodiment of the invention. In plot315, threshold 320 serves as a motion criterion for selecting theinstant of image capture at an approximate local minimum. In oneembodiment, a digital image is captured when the magnitude of motiondrops below threshold 320 (point 330 in FIG. 3B) following activation ofswitch S2 170 at time 325. If the magnitude of motion does not dropbelow threshold 320 within a predetermined timeout period 335, thedigital image may be captured at time 340. The choice of timeout period335 may vary with application or situation, but it would likely notexceed 0.1 second. As an alternative to timeout period 335, the imagemay be captured, despite threshold 320 not being satisfied, if themotion of digital camera 100 is found to be decreasing (in terms ofvelocity, digital camera 100 is decelerating). An example of such aninterval of decreasing motion is that from point 345 to point 350.Anticipating an approximate minimum in this way is particularly usefulin accounting for readout and computational lags in the motionestimation process.

Threshold 320 may be selected based on any of a variety of factors or acombination thereof. In one embodiment, threshold 320 is selected basedon the minimum and maximum motion measured (e.g., minimum and maximumvelocities) during an interval 355 between actuation of switch S1 165and actuation of switch S2 170 (see FIG. 3B). Such minimum and maximummotion measurements within interval 355 are illustrated by points 360and 365, respectively. Threshold 320 may be chosen, for example, as aparticular fraction of maximum 365 or as a value lying between minimum360 and maximum 365. In a different embodiment, threshold 320 may bechosen based on the current focal length setting of optical system 155.A wide-angle focal length of optical system 155 is less sensitive tomotion than a telephoto setting. Therefore, threshold 320 may need to besmaller for a telephoto focal length than for a wide-angle focal length.Likewise, a faster shutter speed also renders motion of digital camera100 less critical. Consequently, threshold 320 may be larger (lessstrict) if digital camera 100 is operating at a fast shutter speed(e.g., 1/500 of a second) than if digital camera is operating at aslower shutter speed (e.g., 1/30 of a second). Another factor that maybe used in selecting threshold 320 is interval 355 in FIG. 3B. Forexample, threshold 320 may be chosen differently depending on whetherinterval 355 is short or long. In yet another embodiment, the thresholdcan be set based on prior characterization of the typical amount ofcamera motion after actuation of S1 165. For example, if a high degreeof motion is detected, due to an unsteady user, one-handed operation,etc., a higher threshold may be selected.

Optionally, threshold 320 may be altered after actuation of switch S2170. For example, threshold 320 may be increased (making the motioncriterion less strict) with the passage of time after the actuation ofS2. This is yet another alternative to timeout period 335 and capturingthe digital image upon detected deceleration of digital camera 100.

FIG. 4 is a flowchart of the operation of digital camera 100 inaccordance with an illustrative embodiment of the invention. If switchS1 165 is actuated at 405, autofocus and autoexposure are performed at410. Once autofocus and autoexposure are complete, digital camera 100enters a motion measurement mode at 415 in which the motion trackingsubsystem measures the motion of digital camera 100 as a function oftime, as explained above. If switch S2 170 is actuated at 420, controlproceeds to 425, and timer 135 may be reset to count timeout period 335.At 425, the output of motion measurement logic 132 is compared withthreshold 320. If the measured motion is less than threshold 320, adigital image may be captured immediately at 435. Otherwise, controlproceeds to 430. At 430, controller 105 checks timer 135 to determinewhether timeout period 335 has expired. If so, the digital image may becaptured at 435. Otherwise, control returns to 425. Once the digitalimage has been captured, the process terminates at 440.

FIG. 5 is a flowchart of the operation of digital camera 100 inaccordance with another illustrative embodiment of the invention. Theprocess in FIG. 5 is similar to that in FIG. 4, except that a decrease(deceleration, in terms of velocity) in the motion of digital camera 100is the criterion for capturing a digital image at 445, when threshold320 is not satisfied at 425.

The foregoing description of the present invention has been presentedfor the purposes of illustration and description. It is not intended tobe exhaustive or to limit the invention to the precise form disclosed,and other modifications and variations may be possible in light of theabove teachings. The embodiment was chosen and described in order tobest explain the principles of the invention and its practicalapplication to thereby enable others skilled in the art to best utilizethe invention in various embodiments and various modifications as aresuited to the particular use contemplated. It is intended that theappended claims be construed to include other alternative embodiments ofthe invention except insofar as limited by the prior art.

1. A digital camera, comprising: an input control to initiate capture ofa digital image; a motion tracking subsystem responsive to the inputcontrol to track motion of the digital camera, an output of the motiontracking subsystem indicating the quantity of motion of the digitalcamera as a function of time; and control logic configured to delaycapture of the digital image until the output of the motion trackingsubsystem satisfies a motion criterion, wherein the motion trackingsubsystem includes an imaging module to convert optical images todigital images, the imaging module being capable of operation in a videopreview mode, the video preview mode producing a series of digitalpreview frames, a portion of each digital preview frame comprising amotion measurement region, wherein the motion tracking subsystem alsoincludes motion measurement logic configured to measure the motion ofthe digital camera by comparing at least one picture element in a firstdigital preview frame with at least one picture element in a seconddigital preview frame, the at least one picture element in each of thefirst and second digital preview frames lying within the motionmeasurement region of that digital preview frame. 2-14. (canceled) 15.The digital camera of claim 1, wherein the imaging module comprises oneof a CCD sensor array and a CMOS sensor array.
 16. The digital camera ofclaim 1, wherein the video preview mode of the digital camera issubstantially the same as that employed during autofocus, and the motionmeasurement region coincides with an autofocus region within whichautofocus is performed on a different series of digital preview frames.17. The digital camera of claim 1, wherein the motion measurement regionof each of the first and second digital preview frames comprises acentral portion of that digital preview frame.
 18. The digital camera ofclaim 1, wherein the motion measurement region of each of the first andsecond digital preview frames comprises a peripheral portion of thatdigital preview frame.
 19. The digital camera of claim 1, wherein themotion measurement logic is configured to apply a window function thatattenuates an edge sub-region of the motion measurement region of eachof the first and second digital preview frames.
 20. The digital cameraof claim 1, wherein the at least one picture element in each of thefirst and second digital preview frames comprises a horizontal set ofpicture elements.
 21. The digital camera of claim 1, wherein the atleast one picture element in each of the first and second digitalpreview frames comprises a vertical set of picture elements.
 22. Thedigital camera of claim 1, wherein the motion measurement logic isconfigured to compute horizontal and vertical motion components.
 23. Thedigital camera of claim 1, wherein the motion tracking subsystemcomprises at least one accelerometer.
 24. The digital camera of claim 1,wherein the motion tracking subsystem comprises at least one gyroscope.25. The digital camera of claim 1, wherein the output of the motiontracking subsystem comprises the square root of the sum of a horizontalmotion component squared and a vertical motion component squared. 26.The digital camera of claim 1, wherein the output of the motion trackingsubsystem comprises the sum of the absolute value of a horizontal motioncomponent and the absolute value of a vertical motion component. 27-38.(canceled)
 39. A method for reducing image blur in a digital camera,comprising: tracking motion of the digital camera in response to a firstinput signal; and delaying capture of a digital image, following receiptof a second input signal, until the motion of the digital camerasatisfies a motion criterion, wherein tracking motion of the digitalcamera includes acquiring a series of digital preview frames in a videopreview mode of the digital camera, a portion of each digital previewframe comprising a motion measurement region; and wherein trackingmotion of the digital camera also includes comparing at least onepicture element in a first digital preview frame with at least onepicture element in a second digital preview frame, the at least onepicture element in each of the first and second digital preview frameslying within the motion measurement region of that digital previewframe.
 40. The method of claim 39, wherein the video preview mode of thedigital camera is substantially the same as that employed duringautofocus, and the motion measurement region coincides with an autofocusregion within which autofocus is performed on a different series ofdigital preview frames.
 41. The method of claim 39, wherein the motionmeasurement region of each of the first and second digital previewframes comprises a central portion of that digital preview frame. 42.The method of claim 39, wherein the motion measurement region of each ofthe first and second digital preview frames comprises a peripheralportion of that digital preview frame.
 43. The method of claim 39,further comprising: applying a window function that attenuates an edgesub-region of the motion measurement region of each of the first andsecond digital preview frames.
 44. The method of claim 39, wherein theat least one picture element in each of the first and second digitalpreview frames comprises a horizontal set of picture elements.
 45. Themethod of claim 39, wherein the at least one picture element in each ofthe first and second digital preview frames comprises a vertical set ofpicture elements.
 46. The method of claim 39, wherein comparing at leastone picture element in a first digital preview frame with at least onepicture element in a second digital preview frame comprises computinghorizontal and vertical motion components.
 47. The method of claim 39,wherein tracking motion of the digital camera comprises measuring themotion of the digital camera using at least one accelerometer.
 48. Themethod of claim 39, wherein tracking motion of the digital cameracomprises measuring the motion of the digital camera using at least onegyroscope.
 49. A digital camera, comprising: means for initiatingcapture of a digital image; means for tracking motion of the digitalcamera responsive to the means for initiating capture of a digitalimage, an output of the means for tracking motion of the digital cameraindicating the quantity of motion of the digital camera as a function oftime; and means for delaying capture of the digital image until theoutput of the means for tracking motion of the digital camera satisfiesa motion criterion.